scenario-notebooks/UserSecurityMetadata/Guided Analysis - User Security Metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
    "source": [
      "# Guided Analysis - User Security Metadata (Public Preview)\n",
      "\n",
      "**Notebook Version:** 1.0  \n",
      "**Python Version:** Python 3.6  \n",
      "**Required Packages**: kqlmagic, validate_email, jsonpickle, azure-cli-core, Azure-Sentinel-Utilities  \n",
      "  \n",
      "**Platforms Supported**:\n",
      "- Azure Notebooks Free Compute\n",
      "- Azure Notebooks DSVM\n",
      "- OS Independent\n",
      "  \n",
      "**Data Sources Required**:\n",
      "- Log Analytics : UserPeerAnalytics, UserAccessAnalytics\n",
      "\n",
      "**Permissions Required**:\n",
      "- **Log Analytics Read Permissions**: To connect and query the workspace you need to be assigned at least [Reader](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#reader) or [Microsoft Sentinel Reader](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#azure-sentinel-reader) role on the workspace.\n",
      "- **Directory Basic Read Permissions** : If you are a user who is a native member of the tenant, then by [default](https://docs.microsoft.com/azure/active-directory/fundamentals/users-default-permissions#compare-member-and-guest-default-permissions) you have permissions to read user, group and serviceprincipal information. If you are a guest user in the tenant, then you need to be assigned [Directory Reader](https://docs.microsoft.com/azure/active-directory/users-groups-roles/directory-assign-admin-roles#directory-readers) role. \n",
      "\n",
      "**Description**:  \n",
      "This notebook introduces the concept of contextual security metadata that are gathered for AAD users. Here are the security metadata that are available* today\n",
      "- **UserAccessAnalytics**: The most important step of a security incident is to identify the blast radius of the user under investigation. This enrichment data calculates for a given user, the direct or transitive access/permission to resources. In Public Preview, we calculate the blast radius access graph only limited to RBAC access to subscriptions.  For example, if the user under investigation is Jane Smith, Access Graph displays all the Azure subscriptions that she either can access directly, via groups or serviceprincipals. \n",
      "- **UserPeerAnalytics**: Analysts frequently use the peers of a user under investigation to scope the security incident. This enrichment data, for a given user, provides a ranked list of peers. For example, if the user is Jane Smith, Peer Enrichment calculates all of Jane’s peers based on her mailing list, security groups, etc and provides the top 20 of her peers. Specifically, this information is calculated using Natural Language Processing algorithms using group membership information from Azure Active Directory.  \n",
      "\n",
      "This is a Microsoft Sentinel **Public Preview** feature. If you are interested in the above analytics data please contact ramk at microsoft com.\n",
      "\n",
      "## Contents:  \n",
      "- [Setup](#setup)\n",
      "    - [Install Packages](#install)\n",
      "    - [Enter Tenant and Workspace Ids](#tenant-and-worskpace-ids)\n",
      "    - [Connect to Log Analytics](#connect-to-la)\n",
      "    - [Log into Azure CLI](#log-into-azure)\n",
      "    - [Enter User Information](#user-input)\n",
      "- [Access Graph of the user](#access-graph)\n",
      "- [Ranked peers of the user](#user-peers)"
    ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='setup'></a>\n",
    "# Setup\n",
    "<a id='install'></a>\n",
    "## Install Packages\n",
    "The first time this cell runs for a new Azure Notebooks project or local Python environment it will take several minutes to download and install the packages. In subsequent runs it should run quickly and confirm that package dependencies are already installed. Unless you want to upgrade the packages you can feel free to skip execution of the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print('Please wait. Installing required packages. This may take a few minutes...')\n",
    "!pip install Kqlmagic --no-cache-dir --upgrade\n",
    "!pip install validate_email --upgrade\n",
    "!pip install jsonpickle --upgrade\n",
    "!pip install azure-cli-core --upgrade\n",
    "!pip install --upgrade Azure-Sentinel-Utilities\n",
    "print('Required Package Installation Complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='tenant-and-worskpace-ids'></a>\n",
    "## Enter Tenant and Workspace Ids\n",
    "You can configure your TenantId and WorskpaceId in config.json file next to the notebook, see sample [here](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/config.json). If config.json file is missing then you will be prompted to enter TenantId and WorkspaceId manually.  \n",
    "To find your WorkspaceId go to [Log Analytics](https://portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.OperationalInsights%2Fworkspaces), and look at the workspace properties to find the ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "import SentinelUtils\n",
    "\n",
    "tenantId = None\n",
    "workspaceId = None\n",
    "configFile = \"config.json\"\n",
    "\n",
    "if os.path.isfile(configFile):\n",
    "    try: \n",
    "        print(f\"Read Workspace configuration from local '{configFile}' file... \", end = \"\")\n",
    "        tenantId = SentinelUtils.config_reader.ConfigReader.read_config_values(configFile)[0]\n",
    "        workspaceId = SentinelUtils.config_reader.ConfigReader.read_config_values(configFile)[3]\n",
    "        print(\"Done!\")\n",
    "        print(f\"Tenant - '{tenantId}' and Log Analytics Workspace - '{workspaceId}' retrieved from {configFile}\")\n",
    "    except:\n",
    "        pass\n",
    "\n",
    "if not workspaceId or not tenantId:\n",
    "    print(f\"Unable to retrive tenantId and workspaceid from '{configFile}'.\")\n",
    "    print('Enter Azure TenantId: ')\n",
    "    tenantId = input().strip()\n",
    "    print()\n",
    "    print('Enter Sentinel Workspace Id: ')\n",
    "    workspaceId = input().strip()\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='connect-to-la'></a>\n",
    "## Connect to Log Analytics\n",
    "This is required to read the tables in your log analytics workspace. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext Kqlmagic\n",
    "%kql loganalytics://code().tenant(tenantId).workspace(workspaceId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='log-into-azure'></a>\n",
    "## Log into Azure CLI\n",
    "Azure CLI is used to retrieve display name and email address of users, groups and service principals from AAD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!az login --tenant $tenantId\n",
    "%run Entities.py\n",
    "%run GraphVis.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='user-input'></a>\n",
    "## Enter User Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Utils import validatedate\n",
    "from datetime import date\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display\n",
    "\n",
    "print('Enter object Id or UPN or email address of the user: ')\n",
    "userIdOrEmail = input().strip()\n",
    "print()\n",
    "\n",
    "if not userIdOrEmail :\n",
    "    raise Exception(\"Error: Empty Object Id or UPN or email address.\")\n",
    "\n",
    "print(f'Retrieving user \"{userIdOrEmail}\" from the tenant...', end = '')\n",
    "user = User.getUserByIdOrEmail(userIdOrEmail)\n",
    "print(\"Done!\")\n",
    "print(\"Name - {0}, Email - {1}, Id - {2}\".format(user.name, user.email, user.objectId))\n",
    "print()\n",
    "\n",
    "print('[Optional] Enter date in format yyyy-MM-dd to retrieve analytics from that date. If you want latest, leave it empty and press enter: ')\n",
    "time  = input().strip()\n",
    "\n",
    "if not time :\n",
    "    today = date.today()\n",
    "    time = today.strftime(\"%Y-%m-%d\")\n",
    "else:\n",
    "    validatedate(time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='access-graph'></a>\n",
    "# Access Graph of the user:\n",
    "Run this cell to visualize the access/permissions of the user in a graph. The cell queries the 'UserAccessAnalytics' table to retrieve direct/transitive RBAC access of the user to subscriptions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output, display, HTML\n",
    "\n",
    "kql_query = f\"\"\"\n",
    "let userId = \"{user.objectId}\";\n",
    "let blastRadTime = todatetime('{time}');\n",
    "\n",
    "let userSubAccess = UserAccessAnalytics\n",
    "| where SourceEntityId == userId and TargetEntityType == \"AzureSubscription\" and TimeGenerated <= blastRadTime\n",
    "| project UserId = SourceEntityId, TimeGenerated , SubscriptionName = TargetEntityName, Subscription = TargetEntityId, Role = AccessLevel, GroupId = \"\", ServicePrincipalId = \"\"\n",
    "| summarize arg_max(TimeGenerated, *) by Subscription, Role;\n",
    "\n",
    "let userGroupAccess = UserAccessAnalytics\n",
    "| where SourceEntityId == userId and TargetEntityType == \"Group\" and TimeGenerated <= blastRadTime\n",
    "| project UserId = SourceEntityId, GroupId = TargetEntityId, TimeGenerated\n",
    "| summarize arg_max(TimeGenerated, *) by GroupId;\n",
    "\n",
    "let userGroupSubAccess = userGroupAccess\n",
    "| join kind = inner\n",
    "UserAccessAnalytics\n",
    "on $left.GroupId == $right.SourceEntityId\n",
    "| where TargetEntityType == \"AzureSubscription\" and TimeGenerated <= blastRadTime\n",
    "| project UserId, GroupId, ServicePrincipalId = \"\", TimeGenerated, SubscriptionName = TargetEntityName, Subscription = TargetEntityId, Role = AccessLevel\n",
    "| summarize arg_max(TimeGenerated, *) by GroupId, Subscription, Role;\n",
    "\n",
    "let userSPAccess = UserAccessAnalytics\n",
    "| where SourceEntityId == userId and TargetEntityType == \"ServicePrincipal\" and TimeGenerated <= blastRadTime\n",
    "| project UserId = SourceEntityId, ServicePrincipalId = TargetEntityId, TimeGenerated\n",
    "| summarize arg_max(TimeGenerated, *) by ServicePrincipalId;\n",
    "\n",
    "let userSPSubAccess = userSPAccess\n",
    "| join kind = inner\n",
    "UserAccessAnalytics\n",
    "on $left.ServicePrincipalId == $right.SourceEntityId\n",
    "| where TargetEntityType == \"AzureSubscription\" and TimeGenerated <= blastRadTime\n",
    "| project UserId, GroupId = \"\", ServicePrincipalId, TimeGenerated, SubscriptionName = TargetEntityName, Subscription = TargetEntityId, Role = AccessLevel\n",
    "| summarize arg_max(TimeGenerated, *) by ServicePrincipalId, Subscription, Role;\n",
    "\n",
    "userGroupSubAccess\n",
    "| union kind=outer userSubAccess\n",
    "| union kind=outer userSPSubAccess\"\"\"\n",
    "\n",
    "print(f\"Executing Kql query to retrieve access analytics for user '{user.name}', on or before '{time}'.. \", end = '')\n",
    "%kql -query kql_query\n",
    "print('Done!')\n",
    "\n",
    "usersubMappings = _kql_raw_result_.to_dataframe()\n",
    "\n",
    "if len(usersubMappings) == 0:\n",
    "    print(f\"No access analytics data available for user '{user.name}', on or before '{time}'\")\n",
    "else:\n",
    "    print('Creating Graph visualization. This may take a few seconds.. ', end = '')\n",
    "    graph = GraphVis()\n",
    "\n",
    "    for index, row in usersubMappings.iterrows():\n",
    "        sub = Subscription(row['SubscriptionName'], row['Subscription'])\n",
    "        rbacRole = row['Role']\n",
    "\n",
    "        if row['GroupId'] == '' and row['ServicePrincipalId'] == '':\n",
    "            graph.addEdge(user.getNode(), sub.getNode(), rbacRole)\n",
    "        elif row['GroupId']:\n",
    "            group = Group.getGroupById(row['GroupId'])\n",
    "            graph.addEdge(user.getNode(), group.getNode(), \"Member\")\n",
    "            graph.addEdge(group.getNode(), sub.getNode(), rbacRole)\n",
    "        elif row['ServicePrincipalId']:\n",
    "            sp = ServicePrincipal.getServicePrincipalById(row['ServicePrincipalId'])\n",
    "            graph.addEdge(user.getNode(), sp.getNode(), \"Owner\")\n",
    "            graph.addEdge(sp.getNode(), sub.getNode(), rbacRole)\n",
    "\n",
    "    print('Done!')\n",
    "    display(HTML(graph.getHtml()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='user-peers'></a>\n",
    "## Ranked peers of the user\n",
    "This cell queries the 'UserPeerAnalytics' table to return a ranked list of peers of the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output, display, HTML\n",
    "import tabulate\n",
    "\n",
    "kql_query = f\"\"\"\n",
    "let userId = \"{user.objectId}\";\n",
    "let snapshotTime = todatetime('{time}');\n",
    "\n",
    "UserPeerAnalytics\n",
    "| where UserId == userId \n",
    "| join kind = inner\n",
    "(\n",
    "UserPeerAnalytics\n",
    "| where TimeGenerated <= snapshotTime and UserId == userId \n",
    "| summarize max(TimeGenerated)\n",
    "| project TimeGenerated = max_TimeGenerated\n",
    ")\n",
    "on TimeGenerated\n",
    "| project PeerUserId, Rank\n",
    "| order by Rank asc\"\"\"\n",
    "\n",
    "print(f\"Executing Kql query to retrieve peer analytics for user '{user.name}', on or before '{time}'.. \", end = '')\n",
    "%kql -query kql_query\n",
    "print('Done!')\n",
    "\n",
    "peerListDF = _kql_raw_result_.to_dataframe()\n",
    "\n",
    "peerList = []\n",
    "peerList.append([\"UserName\", \"PeerUserName\", \"PeerEmail\", \"Rank\"])\n",
    "\n",
    "if len(peerListDF) == 0:\n",
    "    print(f\"No peer analytics data available for user '{user.name}', on or before '{time}'\")\n",
    "else:\n",
    "    print('Retrieving user names and email addresses for peers. This may take a few seconds...', end = '')\n",
    "\n",
    "    for index, row in peerListDF.iterrows():\n",
    "        peerUserId = row['PeerUserId']    \n",
    "        peerRank = row['Rank'] \n",
    "        peerUser = User.getUserByIdOrEmail(peerUserId)\n",
    "        peerList.append([user.name, peerUser.name, peerUser.email, peerRank])\n",
    "\n",
    "    print('Done!')\n",
    "    display(HTML(tabulate.tabulate(peerList, tablefmt='html')))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}